Networking Strategies

  • "Is it better to create a thread for every new connection to the server, considering the connection will block thread execution? Or should I unblock the connection and let every pooling occur within the same thread? How often are both strategies used?"

  • Both strategies are common and have trade-offs. The best choice depends on scale, latency requirements, OS constraints, and language/runtime capabilities.

Thread per Connection

  • Each incoming client connection is handled by a dedicated OS thread.

  • Blocking I/O operations are allowed (e.g., recv() , send() ).

    • Thread sleeps when waiting for I/O and resumes when data is available.

  • Advantages :

    • Straightforward to implement and debug.

    • Each connection is isolated.

    • Uses OS APIs, so no need for complex event loops.

  • Disadvantages :

    • Scalability

      • High memory usage

    • Performance

      • High context-switch overhead

    • OS limits

      • OS-imposed thread limits

  • Use cases :

    • Small-scale servers

    • Systems with <1000 concurrent connections

    • Languages like Java or C++ where thread costs are acceptable

Single Thread + Non-Blocking I/O (with Multiplexing)

  • All connections are handled on one or few threads.

  • Use non-blocking sockets and a multiplexer ( epoll , kqueue , select ).

  • Manual polling or event-driven callback system processes I/O.

  • Advantages :

    • Scalability

      • Supports tens of thousands of connections

    • Efficiency

      • Minimal context-switching

    • Memory

      • Lower per-connection memory overhead

  • Disadvantages :

    • Complexity

      • Manual state machines or async programming

    • Error-prone

      • Harder to debug, manage timeouts, etc.

    • Latency

      • Single-threaded bottlenecks under high load

  • Use cases :

    • High-concurrency servers (e.g., web servers, proxies)

    • Event-driven platforms (e.g., Node.js, Nginx)

    • Languages with async runtimes (Rust/Tokio, Python/asyncio, Java NIO, etc.)

Hybrid Models (Thread Pools + Non-blocking I/O)

  • Use a small thread pool (e.g., per-core or per-socket).

  • Each thread runs an event loop handling multiple connections.

  • Scales well and reduces the risk of single-thread saturation.

  • Common in Go, Netty (Java), libuv (Node.js).